Feed on
Posts
Comments

First impressions of OpenNMS: so far so good

Oct 29th, 2010 by

I got inspired this week to fiddle with OpenNMS. Reading the website makes me think it does * and has great potential. Looking at the modular design and the event system, it just “feels right”. It is Java-based with all XML-based configs, which really doesn’t bother me. I like the idea that it has built in syslog and SNMP trap collectors that automatically associates all active/passive events with the host/node.

Getting started was deceptively simple.  There’s a step-by step guide to installing PostgreSQL and the OpenNMS RPMs. I think I had it running within 30 minutes.  I gave it my home subnet to scan, and it did a great job of providing sensible resource monitoring (with clear, attractive graphs) of discovered things out of the box including OIDs I didn’t know existed. I don’t like network-wide scanning, but I really love the node-based service discovery mechanism. It tries to find every monitorable OID in a list based on the sysObjectId. It also figured out if a system had multiple L3 interfaces, it would do ping/ssh response time monitoring to all public IP addresses. Every single thing it monitored, it graphed automatically, which is a huge annoying shortcoming with Nagios and other monitoring systems.  I’ve recently been evaluating Science Logic’s EM7 and Nimsoft’s solutions, and neither lured me in to get real things done like OpenNMS did.

I’m really disappointed it doesn’t have any IPv6 support. I figured since it’s supposed to be a grown up, modern product that this would already be there.  I’m not even sure if I can hack in anything to do even simple IPv6 ping monitoring even if I wanted to. There’s not that many people talking about support, but looking at the changelogs + single wiki page, support is gradually being added. Now that I’m more familiar with the product, I see that v4 addresses are all over the place and full support will take real work to pull off.

Immediately after it was installed, it felt like they gave me enough rope to hang myself. It was definitely working, but the web interface seemed like it wasn’t the whole story (it’s not). Coming from a Nagios environment, the normal sort of things I’m used to looking at weren’t there or not immediately obvious.  How do I know when a down device will be polled next?  How can I force a re-check?  How can I ignore a check? How can I see all checks on one screen? I basically had to spend five hours one night forcing myself to read over all the configuration how-tos even though a lot of the concepts were foreign to me, then I was able to start constructing a mental model.

Eventually I got comfortable enough with the design and configuration, and started looking for individual things to expand so that it mirrored my Nagios functionality.  I started taking new things one a time, Googling for each one. I was pleasantly surprised I found detailed answers to each thing I wanted so far (most were found on their wiki):

  • How do I monitor every interface (L2 & L3) on a system? Answered on the FAQ.
  • How do I add an Uptime graph to my PIX/ASA firewall?  Add a sysUpTime mibObj under cisco-pix in datacollection-config.xml & adding a section to snmp-graph.properties.xml.  That was it.  It just started working.
  • How do I monitor a running MySQL process?  Here’s a how-to. I kinda wish it would connect directly to the MySQL daemon, but rigging up a web-based stat page is probably a good idea.
  • How do I add UDP/TCP connection rate monitoring to my PIX/ASA firewalls?  Easy as copying from another similar item in datacollection-config.xml and snmp-graph.properties.xml.
  • How do I automatically populate the host asset database via SNMP? Here’s another how-to. This is going to take some work, but doesn’t look bad.

There’s still several things to do before it mirrors what I’ve already implemented in Nagios. One thing I do is change monitoring based on the OS of the device.  Some things like connection rates on PIX/ASA aren’t available until 7.x code. I realized last night that it’s OK to have a OID in a template that’s not supported by the device, such as cufwConnRate1udp on PIX 6.3(5). OpenNMS will figure out the OID doesn’t exist on re-scan and not try to poll that OID ever again.  Another thing I want to do is change connection limit thresholds based on the type of PIX/ASA license, but I don’t know how I’d do this yet.

Troubleshooting Java is annoying.  My first time installing it, I had pages and pages of exceptions and JDBC errors saying it couldn’t connect to the database. It turns out localhost in /etc/hosts was pointing to ::1, so either JDBC didn’t support IPv6 or there was some missing argument. Fixed it back to 127.0.0.1, things started working. That one was truly annoying to figure out.  Last night I was trying to add my ASA connection rate stuff and had 593 lines (wtf!?) of runtime exceptions. The “can’t find datasource” error was completely non-obvious that my datasource name was too many characters long.  I wanted to dump the jRobin RRD files to check them, but the jRobin inspector is X-based and I didn’t immediately find a way to it from a shell.

The RANCID integration is pretty hot. I like having device configuration details automatically available from the OpenNMS web interface.  It took me longer to configure a sample RANCID + viewc (similar to cvsweb) than it did actually turning on the support in OpenNMS. The notification patch for rancid is a nice touch, with it rancid will tell OpenNMS when a config refresh has happened. From the howto, I’m lead to believe that somehow it can manage the .cloginrc password file, but I don’t see how/where this happens. It may not be implemented yet.

So far I really enjoy working with it.  It’s taken literally seconds to add new things to check, things that would’ve taken hours of bailing wire and perl to add to my Nagios install. It’s such great relief that interface/service discovery just works. My RTG targetmaker setup is atrocious, takes hours to run, and not the way to do things. I could easily see this replacing the NMS I hacked together. I haven’t dived into notifications yet, so not sure what’s involved there.

Offhand, here’s my to-do and to-figure-out list:

  • Graphing of packet loss when doing interface pinging. It sounds like StrafePing monitoring is what I want, which is sort of like Smokeping. The docs warn of extra load, I haven’t checked into this.
  • Asset field population from SNMP, including modules/PICs from things like Cisco 6500s, Juniper systems, Cisco CSSes
  • ifStatus monitoring of up/down interfaces
  • BGP peer state monitoring
  • Configure e-mail notifications
  • A more sensible graph report structure, i.e. let me click to drill down to more detail about interfaces, not show me octets/packets/errors/discards together (this becomes fail quickly on switches)
  • Fetching data from SSH (e.g. log in, run a command, parse counters out of a result, store & monitor the data) or even XML. I need to be able to gather tech-support statistics from Cisco CSMs which is only available via CLI.
  • SLB serverfarm/virtualserver stats gathering. Some support for CSS already, but need CSM, Netscaler, and ZXTM too.

It’s interesting that there’s not a lot of blog chatter like Nagios. The bulk of the good information is on the OpenNMS wiki, not much good random info out there. I’m guessing OpenNMS is overkill for a lot of people, and enterprises do their own in-house development to plug into the product. Since it’s all modular and Java, I can easily see it being deeply integrated with other systems and/or do some advanced things. Seeing as how writing my own NMS made me very fluent in perl, working with OpenNMS may finally motivate me to dive in and learn Java.

Tags:

Posted in Uncategorized | 1 Comment »

The future

Oct 25th, 2010 by

The other night I was driving home and in some random apartment window I see two girls with a baby. One was holding/rocking it while she was talking to the other girl.  It made me think that’s a lifetime memory in the making that will be forever repeated to the child. “Oh I remember how I used to hold you when you were a baby and I’d talk about Wife Swap with Jill.”

Tags:

Posted in Uncategorized | No Comments »

IPv6 in the home: handling DNS/name resolution

Oct 21st, 2010 by

If you’re like me, you have a home LAN filled with various contraptions, computers, mobile devices and the ubiquitous server in the closet. Usually there’s no DNS zone containing your home devices, if anything, maybe a static /etc/hosts entry on your desktop to give a short hostname to your closet server. Otherwise, people just use the IP address of the device. Chances are if you’ve setup a home network, memorizing the subnet (e.g. 192.168.130.0/24) is pretty easy, it’s just a matter of knowing that .10 is closetserver, .15 is your XBOX, and so forth. You just “ssh 192.168.130.10” and you’re good.

When devices have IPv6 addresses, using the IP address isn’t feasible anymore. There has to be some sort of name lookup mechanism. It’s just too long to remember an IPv6 address and there are fiddly things to watch for. Nobody is going to run a real DNS server in their house either.

In the simplest case of using only link-local addresses, you’re just not going to remember your closetserver is fe80::216:e6ff:fed8:d122. You can put a host entry on your desktops, but this involves updating everything by hand. Link-local addressing also presents an additional challenge in that if you have a computer with multiple interfaces (think: laptop with a gigabit LAN interface and a 802.11 wireless interface), you have to specify the interface to use in your /etc/host entry, i.e.

fe80::216:e6ff:fed8:d122%en1  closetserver

This would tell your application to use the en1 (wireless) interface to get to closetserver.  But, if you turn off your wireless and plug into your switch, all of a sudden en1 is the wrong interface to use. Now you have to go edit /etc/hosts again to change from %en1 to %en0.

A slightly more advanced example is if your home network uses your router (in my case an Apple Airport Extreme) as a 6to4 gateway.  The IPv6 address is still to long to remember, but now the network prefix can change based on your externally visible IP address.  Today closetserver may be 2002:4770:2e84::216:e6ff:fed8:d122. Tomorrow your cable modem may reset, giving you a different dynamic IPv4 address and thus a different 6to4 prefix. Now closetserver’s address has changed to 2002:4778:1156::216:e6ff:fed8:d122, and you need to go update all those /etc/host entries.

Solution: Multicast DNS (mDNS). Also known in some documents as “serverless DNS”.

It turns out there’s at least already one solution for all this running behind the scenes that I wasn’t aware of, Multicast DNS. This part of what Bonjour/Zero-conf networks are built on. Each device joins a multicast group on your LAN, advertising and responding to name lookups without contacting an external DNS server or /etc/hosts.

Almost all Apple OS X and iOS devices already have Multicast DNS (mDNSResponder) running, which is how things like iTunes sharing and streaming work, and lets me access resources on the network via name such as “firecracker” for my desktop without knowing the IP address. On Linux, this is handled by Avahi. On Windows, it’s Bonjour for Windows.

This isn’t strictly a problem with just IPv6 either, and there are probably other things out there that do this. This is most interesting to me because the majority of my devices are Apple products.

Here’s how it works:

From my laptop, I want to ssh to closetserver (which is running Avahi). In order to use mDNS, I need to use a “.local” suffix:

lapdance:~ bwann$ ssh -6 closetserver.local

My laptop queries the mdns multicast address ff02::fb on port 5353 asking anyone for a AAAA record of closetserver.

16:01:20.585290 IP6 lapdance.local.mdns > ff02::fb.mdns: 0 AAAA (QM)? closetserver.local. (30)

The mDNS responder on closetserver responds to the multicast group with the AAAA record to use.

16:12:12.098910 IP6 2002:4770:2e84::216:e6ff:fed8:d122.mdns > ff02::fb.mdns: 0*- [0q] 2/0/0 (Cache flush) AAAA 2002:4770:2e84::216:e6ff:fed8:d122 (80)

Then the ssh program initiates the usual TCP with closetserver and we’re done:

16:16:33.518129 IP6 2002:4770:2e84::216:cbff:fe09:a76e.60425 > 2002:4770:2e84::216:e6ff:fed8:d122.ssh: Flags [S], seq 832255505, win 65535, options [mss 1440,nop,wscale 2,nop,nop,TS val 1053744031 ecr 0,sackOK,eol], length 0

Now if I change from wireless to wired, change my external IPv4 address, or even use a new desktop, I can still use the same hostnames without changing anything.  Neat!  This would also be useful for ad-hoc networks as well.

Men & Mice has a great presentation describing Multicast DNS.

Tags: ,

Posted in Uncategorized | No Comments »

The 1960s

Oct 17th, 2010 by

Normally I don’t think a lot about American culture, but the other day I realized just how profound the 60s was on America and the world. I’ve been listening to a Beatles station on Pandora for a couple of weeks, and the other day while working I was keeping an ear tuned to the lyrics as songs played. A couple of the songs were about Vietnam and the draft, and it got me thinking about all the events in the 60s as a whole.

On the surface, to me it has always been “oh yeah yeah the 60s”. I knew individually the Cold War, Vietnam War, the space race, the Cuban Missile Crisis, JFK, Elvis, Beatles, hippies, communism fear, and student demonstrations happened in the 60s. Not to mention the spectacular volumes of now-legendary music that was made then. But thinking about all these events as a whole on the same time line, along with the new civil rights laws and drafts made me realize that there was a lot of permanent change going on. We conscripted men to war. We went to the moon. All within a decade. People and a nation were motivated.  It seems hard to me that anyone couldn’t get caught up in it all. Anti-war protests not your thing?  Then get on board cheering on the Apollo missions! Maybe go to Berkeley to make/experience cool music instead. Watch some Star Trek!

In this way, it reminds me of World War II. It’s easy to think about all the fronts/theaters as discrete wars. You had the Germans moving around Europe, the Russians defending their cities, U-Boat attacks in the seas, then across the globe we were fighting the Japanese. But, all these events happened within a several years of each other in the 40s and were definitely related.

I can’t name many fascinating things that the 70s and 80s gave to us.  Porn and energy crises?

Posted in Uncategorized | No Comments »

1 week: Redmond vs Austin

Oct 17th, 2010 by

It’s been a full week since I moved into my own apartment. It’s starting to feel more like I’m grounded now instead of being on four vacations for the past five weeks. Pretty much everything is put away or tossed in the bed-less bedroom. I’m trying hard not to buy all new furnishings from IKEA, but I tell myself the stuff I buy isn’t /that/ obvious or tacky.

I also got internets through Verizon DSL on Wednesday, so I’ve been working from here since. It feels beneath me to get my internet through a pair of copper, but it’ll have to do until FiOS is available. My work seating has upgraded from a foam Thermarest pad, to a pair of milk crates, now to a stack of Pelican cases.

I’ve built up a small list of things that I’ve already noticed that are different than what I’m used to compared to Texas that really stood out so far:

  • Everything is so close together. It feels like a big small town. The grocery store and Trader Joe’s is two blocks away. I don’t even have to venture outside to get to my truck, check my mail, or take out the trash.
  • It’s also quiet and dark in downtown Redmond at night, not much going on. The streetlights seem to only be in places where they’re necessary such as corners and crossings. I realized the other night that there’s virtually no stars at night from the cloud cover. And, there’s no giant mega car lots that pollute the horizon with light.  There’s also no “party street”. I haven’t decided if I like this or not, that maybe I’m missing the scene in Seattle. Then again, parking is easy on the eastside.
  • Iced tea is hit or miss. Many places sell tea made from Nestea mix, but will usually at least warn me first.
  • I’ve already had more unique visitors here in the past week than I have the past 2 years in Austin.
  • I-5 and I-405 are usually congested so I have to re-adjust what “only 15 miles” means to me. In Austin I thought nothing of driving 15 miles to Homeslice Pizza because I could drive fast on MoPac. Here that can turn into a 45 minute trip.
  • There’s a surprising amount of Mexican (what I would consider as Tex-Mex) in Redmond.
  • The homeless are hardcore here. The other day I saw a guy with super long, greying hair, in denim shirt + pants, soaking wet and shivering from the rain, holding up a waterlogged cardboard sign that was falling apart. They is also no shortage of loud obnoxious ones in downtown Seattle.
  • Even going to IKEA is weird. They have two giant warehouses converted into indoor parking lots.
  • Lack of central head & air is weird. Each room is a different temperature, the air can get stuffy, and mold is a real risk here.
  • I no longer have to subtract 2 hours when I think about people on the west coast. Now I have to add 2 hours when thinking about Oklahoma or Texas.
  • Peet’s Coffee is here, so delicious. I’ve always bought my tea online from them, it was complete accident that I found their store.
  • Fred Meyer is huge. I can’t comprehend it in order to describe to somebody else. It feels like an uppity Target with the food section from a Wal-Mart Supercenter, but not quite.
  • Trader Joes is here too, I love buying food here and look forward to trying more.
  • Rent is a bit more expensive, food is a couple dollars more, gas is way more, groceries seem the same.
  • When I’m indoors I don’t notice the weather. It’s usually not until I go outside I realize it is raining or cloudy.
  • So far there’s been quite a bit of sun here. Everyone has been taunting me to enjoy it while it lasts.

Tags:

Posted in Uncategorized | No Comments »

Send all traffic over VPN to a Cisco ASA

Oct 11th, 2010 by

Let’s say you’re operating on sketchy wifi and the quiet guy in the corner on his laptop is sniffing your traffic. Web is already easy, you fire up SSH port forwarding and tell your browser to use yourself as a proxy. Other apps aren’t so easy. Not everything supports/honors SOCKS5 proxies, or any sort of proxies for that matter.

From my tcpdumps over the weekend, despite having various remote proxies (via v4/v6 AnyConnect VPN) configured in the System Preferences pane and FoxyProxy, there were several things that try to hit the Internet directly. Some background Facebook actions, Mobile Me, instant messaging apps, and Pandora (via PandoraJam) were leaking to the outside. DNS was also unprotected. Some of the leak was SSL traffic, but the majority was all clear-text. Yahoo! IM claims to try to use a SOCK5 proxy via the MacOS’s web proxy preferences, but apparently it falls back to using the public internet. There’s no way to know when it does/doesn’t use a proxy.

It took me a little while of hunting to figure out how to do this “properly” on an ASA. Most examples try doing this with a split-tunnel ACL that works on 0.0.0.0/0. It turns out that Cisco has a document specifically for this  and they call the method “VPN Client for Public Internet on a Stick” (of course).  Under  a group-policy there is a “split-tunnel-policy tunnelall” directive instead of “tunnelspecified“.  Don’t forget your global NAT!

Tags: ,

Posted in Uncategorized | No Comments »

Homeless no more!

Oct 7th, 2010 by

I finally found & leased an apartment yesterday! It’s a 2 bedroom, 1 bath, located in a mid-rise building in downtown Redmond. It’s a corner unit on the second floor, one side overlooks the courtyard and the other overlooks the street.  I originally went to look at a 1 bedroom, which I didn’t like because there was practically no storage for all my fuh.  I wound up with a 2 bdr because the leasing girl happened to mention the cost, it was still within my price range and was a great deal.  The internets give poor ratings to this place, the more interesting review claims there was water leaking out of an electrical outlet from upstairs. So, we’ll see how things go for me.

Pretty much all last week I was in NJ/NY for work. I flew out Tuesday, sat in meetings all day Wednesday and Thursday, worked from Somerset on Friday, and flew back on Saturday. I found out that flying from coast to coast is a great way to eat a day. I left for SEA at 5:20 AM Pacific, was in the air for five hours, landed at EWR, then it was about 5:30 PM Eastern. My coworkers gave me the grand tour of Hoboken, NYC, Somerset, Jersey City and Newark. Even at 4 PM on a Friday, downtown Newark was sketchy. I can’t imagine it at night. Wednesday night a vendor took us out to Old Homestead Steak House. While the steak was great, it was the mac n cheese that was really outstanding.

While in NJ we used car services to get around. I can’t forget our awesome driver on Thursday morning. He said he had moved from Texas 27 years ago, worked as a professional chef, got tired of cooking and took up driving. He wasn’t afraid to put the pedal down and put the Lincoln Navigator through traffic. He says he owned a Ferrari, but gave it up when the shop wanted to do a $27k overhaul on it. In addition, he told us how way back when, him and his wife bought a 800 sq/ft house + four acres for $225k. Through a combination of taking on roommates, saving, and gradually adding on to their house, he now has a 12,000 sq/ft house on the same four acres that would “easily go tomorrow for $6.3 million”. Once he finishes putting his last child through college, he’s moving to Texas to retire, buying a lot of land, and must have a Hummer, another Mercedes, and some sort of sports car. Clearly I’m doin’ it wrong.

This week I managed to get sick with a cold and sinus congestion. Today I’m a little coughy, but much better than yesterday. As soon as I could tell I was getting sick, I went out and bought another neti pot. I wanted to flush out what I could before it ate up my throat and made things worse. This seems to helped, as I don’t have an irritated throat like I normally would.

Now I need to find some movers to take all of my stuff from storage to the new place. I’m sad there’s no FiOS yet (“a month or two”) and Comcast won’t have my internets ready until next Thursday at least.

Tags:

Posted in Uncategorized | 1 Comment »

Get IPv6 on your iPhone via AnyConnect

Sep 30th, 2010 by

I discovered today there’s a way to bring IPv6 connectivity to your iPhone, even if you don’t have v6 wifi nor v6 cellular data.  There’s a Cisco AnyConnect client for iPhone which speaks SSL VPN (TLS/DTLS) to an ASA.  The release notes say “Access to internal IPv4 and IPv6 network resources”. I take this to behave just like the standard AnyConnect client. Unfortunately it requires a AnyConnect Essentials license for the ASA which is a couple hundred dollars.  The client for the iPad should be available whenever iOS 4.2 comes out.

Sep 30 2010 16:31:02: %ASA-3-716057: Group <webvpn> User <bwann> IP <23.131.43.173>
  Session terminated, no AnyConnect Mobile license available

AnyConnect for Mobile          : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials          : Disabled

http://itunes.apple.com/us/app/cisco-anyconnect/id392790924?mt=8#

Tags: , ,

Posted in Uncategorized | No Comments »

IPv6 over IPv4 using Cisco AnyConnect DTLS VPN + ASA

Sep 27th, 2010 by

My host’s home network is behind NAT, has no 6to4 connectivity and the router squashes IP protocol 41 (6in4). Normally in this situation I’d fire up a Teredo tunnel, but here it’s pretty unreliable.  My control traffic to Microsoft’s Teredo server goes to Singapore, and who knows where my closest Teredo relays are. After some random period of time the tunnel is unusable and comes back later.

At first I tried using Hurricane Electric’s PPTP VPN to tunnel my 6in4 traffic over a PPTP VPN from my Mac desktop and laptop, but never got either one to work. I could send traffic outbound, see it received on the remote host, but never see the return traffic.  Interestingly, if I initiated traffic from the remote to my desktop, I saw traffic in both directions.  I’m not a smart person, so I gave up and resorted back to Teredo.

My lab network has full IPv6 connectivity, so my next thought was trying to run an AnyConnect-based VPN which claims to allow IPv6 over IPv4. This would allow me to carry my v6 traffic over an SSL VPN (DTLS) over 443/TCP, which avoids the whole clobbering of proto-41 traffic and shoddy Teredo tunneling.  I figured out while a PIX 515 with 8.0 works fine for basic IPv6 firewalling to hosts, one needs an actual ASA to terminate AnyConnect clients.

I configured an ASA 5505 running 8.2(3) on my test network. I configured WebVPN/AnyConnect for SSL VPN (SVC), then came back and added some IPv6 VPN bits. There actually wasn’t much to do, and it’s decently documented in the AnyConnect admin guide. The “IPv6 Tunnel Default Gateway” seems unneeded and took forever to figure out what exactly it did: secondary/fallback static route to in case you want to send traffic to another router instead of the ASA.

ipv6asa# show vpn-sessiondb svc
Session Type: SVC

Username     : bwann                  Index        : 21
Assigned IP  : 10.255.254.2           Public IP    : 98.117.25.184
Assigned IPv6: 2600:c1f0:0:f00e::6
Protocol     : Clientless SSL-Tunnel DTLS-Tunnel
License      : SSL VPN
Encryption   : RC4 AES128             Hashing      : SHA1
Bytes Tx     : 15450063               Bytes Rx     : 3679785
Group Policy : webvpn                 Tunnel Group : webvpn
Login Time   : 00:18:49 UTC Tue Sep 28 2010
Duration     : 4h:38m:49s
Inactivity   : 0h:00m:00s
NAC Result   : Unknown
VLAN Mapping : N/A

It works quiet nicely. same-security-traffic permit intra-interface enables hairpinning, allowing me to access outside v6 resources from my VPN connection. Of course there’s not much feature parity here. There’s no IPv6 split tunneling that I can tell, it just assigns an address from the pool to my workstation and sets my inet6 default route at it. My v6 pool is global-scope space that’s routed to my ASA.

If your hosting network doesn’t have outside v6 connectivity, you can still use this to communicate with servers behind your firewall via v6. You can get away with using a site-local FEC0:: network as your pool. But, this is borrrrrring.

Tags: ,

Posted in Uncategorized | No Comments »

Redmond limbo

Sep 27th, 2010 by

After my great journey I arrived in Redmond Saturday-before-last (18th). Saturday night was a nifty welcome-to-the-PNW get-together  at Coppola-Leigh. Sunday I rented a storage unit and unloaded all my belongings. Thanks to Alex & Victoria for coming to help unload awesome heavy couch and the rest of my boxes!  I’ve taken residence in the dining room, having taken over the coffee table for work.

The first few days felt like a surreal anti-vacation. I felt like I was only here temporarily, but knew I wasn’t.  After unloading I realized my routine was gone. Several thoughts ran through my mind, such as “man, a Sonic iced tea sounds awesome” but there was no Sonic.  “Man, I’m hungry, Chuy’s sounds awesome” but I didn’t know where any good restaurants were.  “Man, I just want to go home and put my feet up” but I had no home to go to.

I’m still looking for an apartment. Some I looked at last week were either way expensive or didn’t have any units available until November. Tomorrow I leave for EWR for the rest of the week, so that puts a hold on visiting properties.

There are a few good stories of events that happened along the way and after I got here. I need to write about these later!

Tags:

Posted in Uncategorized | No Comments »

« Newer Posts - Older Posts »