While in Redmond, I’ve discovered two delicious sources of food. One is the medium especial salsa and restaurant-style tortilla chips from Trader Joe’s. It’s a great mix of heat and salty, mmm. The other is the coconut curry from Tandoori Fire just down the street. I’ve been trying new Indian places lately and have re-discovered their curry. It’s so rich and creamy, and is amaaaazing with garlic basil naan.
Hard lessons learned about OpenNMS database maintenance
Jan 10th, 2011 by bwann
I learned a hard lesson in PostgreSQL administration, in particular what happens when tables aren’t vacuumed. I come from a MySQL background and haven’t fully learned what it takes to keep Postgres humming along. I had recently moved the OpenNMS database to another host in an attempt to throw more spindles at a UI slowness issue. Most everything else worked well except the events page would time out while trying to render. Moving Postgres to another box worked well until it started getting slow again over the holidays.
Looking at my resource graphs for the database server over the previous week, the amount of CPU time spent in I/O wait grew considerably until it eventually stayed pegged out doing nothing but read operations from the disks. This didn’t seem “right” since I hadn’t added any new switches and the configuration was the same. Since this was some random box with 2 GB of RAM and I had really been generous with the memory allocations to PostgreSQL, I thought maybe I was exhausting the Linux buffer cache as it was hovering around 6700 KB. Tuning down Postgres’ memory footprint, it didn’t affect buffer cache and I was still doing a ton of read ops off disk.
I fixed the pgsql stats collector and went hunting for what OpenNMS was doing to generate so many reads. It turns out that an UPDATE to the snmpinterface table was consistently taking several seconds to execute. This is where I learned that PostgreSQL tables need to be vacuumed frequently, especially ones where there are many UPDATEs on that table. Even after stopping OpenNMS and letting the box do its thing, it took several hours for a ‘VACUUM VERBOSE snmpinterface’ to finish. Even though I had auto_vacuum turned on, it wasn’t actually working because “stats_row_level” wasn’t set to on. Therefore, the tables were never vacuumed!
While that was running, I was reading the event maintenance documentation. I checked out the events table and discovered it too had become bloated. I update the provisioning group hourly and I had hundreds of thousands of ‘nodeScanCompleted’ and reinitializePrimarySnmpInterface’ UEIs in my events table. Since I really don’t care about these events, I deleted them and added a statement to vacuumd-configuration.xml to purge them after 2 days. This made a gigantic improvement in displaying events in the web UI, namely, it actually worked again.
Between cleaning up both tables and fixing vacuuming, the excessive read operations completely vanished. From what I can tell, because of the way PostgreSQL’s transaction stuff works, any time it did an update it was having to read all these old transactions off disk first and over time it was a vicious feedback loop.
Lessons learned:
sar -d 1 300is your friend. If “rd_sec/s” is stays around zero, that means your OS filesystem cache is adequate and it’s not having to go to disk frequently for hot blocks. Finally realizing how VM management and buffer cache works in practice in Linux was a great experience. This is something one of my old bosses tried to teach me a long time ago and it never really sunk in until now.- PostgreSQL’s stat collection tries to connect to a random high-numbered port on 127.0.0.1. If you have deny-all, permit specific rules like I do, make sure you accommodate this with “-A INPUT -i lo -j ACCEPT” or something. Otherwise, the pg_stat tables are empty and there’s no indication why.
- The I/O characteristics between the database and RRD storage are complete opposites. The RRD filesystem deals with tons of tiny write operations. The database server does a modest amount of reading large chunks of data off disk (table scans?) with relatively few writes.
- Keep the events table trimmed, apparently there’s a lot of event correlation going on in the background and extra rows can really slow it down.
- Check pg_log/postgres-X.log to make sure auto vacuuming is actually working.
Now to see what the next learning experience is! If I can figure out what’s causing all the “storeResult: org.opennms.netmgt.snmp.SnmpResult” to be written to my output.log, I’ll be happy. (changing collectd logging to FATAL hasn’t seemed to help)
California New Years
Jan 5th, 2011 by bwann
Bay Bridge at night from Treasure Island
For New Years, I had Thursday and Friday off work. I haven’t been on a good road trip since moving, and I was itching to go explore and see some sun. I had the loose idea of driving to SF, then following the 101 back up the coast since I’d only covered half way long ago. Thursday morning at 10:30 I headed south.
It was a cold, but really sunny drive through Oregon. There were several valleys topped with fog that I drove down into. Inside the valleys, everything was covered in frost and the trees had snow on their branches. The drive went amazingly fast, I was excited about going to the Bay. I arrived at Berkeley around 10:40, almost exactly 12 hours later. I kept on driving across the Bay Bridge and went to the Golden Gate Bridge (after momentarily getting lost downtown trying to follow the 101).
The sky over the bay was nearly perfectly clear and the stars were amazing. I don’t know if I’ve ever been to the GG park at night, much less on a clear night. I stayed out until 2 or 3 AM taking long exposure star photos. I was having a good time but I was absolutely freezing. Wearing two of my mountaineering coats and hat, I resorted to using my iPhone as a hand warmer by firing up the GPS app and letting it burn CPU cycles.
I drove back to east bay to find a hotel for the night. I landed at a hotel in Berkeley around 4:30 AM. The clerk was nice enough to consider my late arrival as an early arrival for Friday, essentially giving me a free day. yay! I was having an absolute blast driving around and I ditched my plan of leaving to drive up the coast.
East of Telegraph Ave in Berkeley
I spent little time in my hotel, instead spending a lot of time walking around Berkeley and driving around the Bay. The weather was similar to Seattle but much nicer and slightly warmer. The times it was cloudy, it was actually dry. I forgot what dry ground and pavement was like! There were still periods of rain, but it was generally over quick. Most everyone I talked to said it was odd that it was the weather was so cold. Fine by me!
I didn’t know the Bay gets all crazy with fireworks for New Years. Had I actually thought to research it, I would’ve planned different. Instead I was driving back from Oakland, tired with the cold and drizzle. I got back to my room just in time to hear fireworks going off at midnight everywhere but not able to see them anywhere. After a few luls I finally decided to drive up to Tilden to see if I could see anything. Unfortunately it was still rainy and the overlooks by the park were really too far away to see the small stuff going off (and I got lost). Apparently the right thing was to go to Berkeley Marina to see the fireworks. Next time, maybe.
There was no plan at all. I visited some friends, ate lots of food, drove all around the Bay, took lots of photos, and sat on the internet at Peet’s. I was pretty excited and glad to be back in the Bay. It seemed like there was an energy I was picking up on, and it was really nice. I definitely want to go back and spend more time there. I want to visit the UC Botanical Garden (the foothills were super pretty) and should spend more time in SF proper.
I was disappointed to leave, but it was time to go back to work and I was tired of living in a hotel. The drive back went fast, I only had to stop three times for 10 minute cat naps.
PNW darkness
Dec 22nd, 2010 by bwann
I gotta move. The winter solstice was yesterday and the sun just set behind the trees at 3:47 PM. It doesn’t help that Redmond is in a valley so it sets early. I made the 405 – 90 – Seattle – 520 – Redmond loop this afternoon to soak in the sun while it was out. The darkness and constant work from home is driving me crazy. I’ve been out every single night for the past five days, just to go elsewhere. I leave for Oklahoma in the morning and I’m already pondering going to the U district this evening to wander.
I haven’t been on any good road trips since moving, and I just discovered how cheap it is to fly down the coast. I need to make some trips to the SF bay and LA to go exploring.
More OpenNMS
Nov 30th, 2010 by bwann
OpenNMS has a sharp learning curve. While very useful in specific cases, the OpenNMS wiki lacks continuity and in some cases authoritative answers. It seems like every article is where somebody worked out the details of some specific problem and just dumped all the details into a wiki entry. The in-between bits aren’t clear and take a good amount of detective work to figure out. I’ve seen several articles written in the form “well it looks like OpenNMS does X, I guess they intended for it to do foo”, as if there’s no maintainers to help clarify.
The really useful information I found was in the few whitepapers, case studies and OpenNMS Conference presentations. Even if some of the papers are in German, the included screen shots and configuration examples are still in English and still provide useful ideas.
These are the things on my wish list/todo list and what I’ve figured out so far:
- There’s two completely different processes that use SNMP to talk to a host. One just checks to see if SNMP is alive by fetching sysObjectId. The other process is the actual data collection that winds up written to disk.
- Measuring packet loss to a device: use StrafePing
- Interface status monitoring (ifAdmin/ifOper): use the “SNMP Service Poller”.
- BGP peer state monitoring: The approach in “BGP Session Monitor” is annoying, and there’s no better way to do it because of the system design. You have to manually configure a capsd + poller item for each and every BGP session you have. I deal with hundreds of peering sessions, I want to know when they go down, and this gets unwieldy. This requires a separate external script to build the configuration. Fortunately I already have a script to generate the BGP configuration for Nagios that’s easily adapted. Ideally I think there should be a capsd process to find all of these like physical interfaces.
- Cisco module monitoring: same thing as BGP monitoring, you have to have an external script to figure out what modules are installed and configure the appropriate capsd+poller files. It may just be more useful to focus attention on catching traps instead of trying to poll status+deal with traps.
- Having OpenNMS run an external command (i.e. SSH) to gather data: don’t. I figured out this really is possible, but the Right Way is to abstract the exotic data collection so that OpenNMS just fetches a web page with the data to be monitored and doesn’t care about the details of how those data were collected. This abstraction will help even if you change to another graphing package such as Cacti.
- Recording serial numbers from devices: “SNMP asset adapter”. You’ll need to add configuration to map data from say, Cisco entPhysical, to the various asset fields like “description” or “serialNumber”. Unfortunately there’s no good way to clearly document serial numbers/hardware information of installed modules in say a Cisco 6500, Cisco CSS or a Juniper router.
- Automatic L2 topology discovery with Linkd. This blew my mind at how well it worked out of the box (it’s disabled by default). By default it collects CDP, bridge information and route tables from hosts, then provides a hypertext list of what’s connected to what on the web interface. I’m sticking to only using CDP collection on my switches, as I have hundreds of VLANs and thousands of routes, and I’m afraid of what this will do to switch CPU.
Tips & ideas:
- Interface status: Sometimes we just don’t care about interface status, or we need temporary exclusions (i.e. new turnup, testing, laptop ports) and don’t want to be constantly annoyed by alerts. Edit snmp-interface-poller-configuration.xml, change the interface criteria to include “and snmpifalias not like ‘%IgnoreStatus%” (attentive users will notice this is really SQL). Now when OpenNMS sees “IgnoreStatus” in the description, it’ll ignore it.
- Interface status #2: Sometimes we only care about port-channels and uplinks on switches, and not care about ports facing servers. Using the same logic above, you can configure the poller to only monitor physical interfaces with like “Uplink” or “Port-channel” in the description, ignoring everything else.
Unanswered questions:
- Does SNMP4J or OpenNMS try to aggregate collection of OIDs? Is it possible to exclude OIDs? In my SNMP data collection config, I’m trying to fetch hrSystemUptime, hrSystemUsers and hrSystemProcesses. Either OpenNMS or SNMP4J (I think the latter) tries to be smart and causes it to fetch the entire hrSystem table. This is a problem because I have some systems where fetching the whole hrSystem (1.3.6.1.2.1.25.1) tree causes the agent to stop responding to requests, consequently generating repeated SNMP timeouts. This is very reproducible and masking the extra OIDs in Net-SNMP doesn’t seem to help. Upgrading Net-SNMP hasn’t helped either. While Net-SNMP is probably the root cause, there’s no easy way that I can see to have OpenNMS to stop trying to collect too much data. Surely there’s enough buggy agents out there that people have solved this.
- Why does OpenNMS generate duplicate interface graphs on new nodes? Upon addition of a new node, it creates directories for RRD/JRD files using the interface name such as “Gi3/13”. Later, after a service scan happens, it apparently figures out the MAC address and creates a new directory using the interface name + MAC address such as “Gi3/13-0e63cafe153”, orphaning the old interface data. Why can’t it do this first?
I’ve been neck deep fiddling with this stuff and it almost feels like I could write a book about it.
Redmond weather
Nov 21st, 2010 by bwann
The changing season and weather here has been a trip. For the first month or two here, it was partially cloudy, but still quite a bit of sun. Even though it’s sunny, being outside has always felt weird. It took me a while to realize that because the sun is so low on the horizon, long shadows were being cast off of everything, even at noon. This made it feel like it was much later than it really was.
After daylight savings ended, the long darkness started setting in. The sun goes down behind the trees around 4:30 PM. The week and a half have been very cloudy, thick clouds at a low altitude. I realize now why people say the sun comes out at the end of the day: it’s finally low enough on the horizon to shine under the cloud layer. This usually lasts 15-30 minutes before it sets. There’s still a month of shorter days ahead. I went out to Target and bought several more lamps to keep the apartment bright.
I had the obvious, expected reaction to the clouds. I started feeling gloomy when I couldn’t even tell where the sun was for a couple of days. What I wasn’t expecting was how giddy I got when I could see the sun again. The same goes for seeing the moon + stars too. It’s fairly dark in Redmond, there’s no excessive streetlamps, and the clouds blanket out the sky completely. The other night I walked to Trader Joe’s and noticed the sky was clear. I could see the moon and quite a few stars. The sky felt like a dark version of Montana big sky, just so expansive and big!
Today it finally snowed! I was kind of disappointed it started off sunny, then got cloudy. I wasn’t expecting it to snow, but it did. This made me terribly excited. I drew all the blinds and sat in my office with my feet propped up on the windowsill watching it snow. After a while the flakes got bigger, so I used TJ’s as an excuse to go walk around in it. That was about 30 minutes ago, it’s since stopped snowing. :(
I need to plan my sun retreats. I learned today that the passes may be closed to non-4x4s and those without chains, so this limits the domain to north/south.
First impressions of OpenNMS: so far so good
Oct 29th, 2010 by bwann
I got inspired this week to fiddle with OpenNMS. Reading the website makes me think it does * and has great potential. Looking at the modular design and the event system, it just “feels right”. It is Java-based with all XML-based configs, which really doesn’t bother me. I like the idea that it has built in syslog and SNMP trap collectors that automatically associates all active/passive events with the host/node.
Getting started was deceptively simple. There’s a step-by step guide to installing PostgreSQL and the OpenNMS RPMs. I think I had it running within 30 minutes. I gave it my home subnet to scan, and it did a great job of providing sensible resource monitoring (with clear, attractive graphs) of discovered things out of the box including OIDs I didn’t know existed. I don’t like network-wide scanning, but I really love the node-based service discovery mechanism. It tries to find every monitorable OID in a list based on the sysObjectId. It also figured out if a system had multiple L3 interfaces, it would do ping/ssh response time monitoring to all public IP addresses. Every single thing it monitored, it graphed automatically, which is a huge annoying shortcoming with Nagios and other monitoring systems. I’ve recently been evaluating Science Logic’s EM7 and Nimsoft’s solutions, and neither lured me in to get real things done like OpenNMS did.
I’m really disappointed it doesn’t have any IPv6 support. I figured since it’s supposed to be a grown up, modern product that this would already be there. I’m not even sure if I can hack in anything to do even simple IPv6 ping monitoring even if I wanted to. There’s not that many people talking about support, but looking at the changelogs + single wiki page, support is gradually being added. Now that I’m more familiar with the product, I see that v4 addresses are all over the place and full support will take real work to pull off.
Immediately after it was installed, it felt like they gave me enough rope to hang myself. It was definitely working, but the web interface seemed like it wasn’t the whole story (it’s not). Coming from a Nagios environment, the normal sort of things I’m used to looking at weren’t there or not immediately obvious. How do I know when a down device will be polled next? How can I force a re-check? How can I ignore a check? How can I see all checks on one screen? I basically had to spend five hours one night forcing myself to read over all the configuration how-tos even though a lot of the concepts were foreign to me, then I was able to start constructing a mental model.
Eventually I got comfortable enough with the design and configuration, and started looking for individual things to expand so that it mirrored my Nagios functionality. I started taking new things one a time, Googling for each one. I was pleasantly surprised I found detailed answers to each thing I wanted so far (most were found on their wiki):
- How do I monitor every interface (L2 & L3) on a system? Answered on the FAQ.
- How do I add an Uptime graph to my PIX/ASA firewall? Add a
sysUpTime mibObjundercisco-pixindatacollection-config.xml& adding a section tosnmp-graph.properties.xml. That was it. It just started working. - How do I monitor a running MySQL process? Here’s a how-to. I kinda wish it would connect directly to the MySQL daemon, but rigging up a web-based stat page is probably a good idea.
- How do I add UDP/TCP connection rate monitoring to my PIX/ASA firewalls? Easy as copying from another similar item in
datacollection-config.xmlandsnmp-graph.properties.xml. - How do I automatically populate the host asset database via SNMP? Here’s another how-to. This is going to take some work, but doesn’t look bad.
There’s still several things to do before it mirrors what I’ve already implemented in Nagios. One thing I do is change monitoring based on the OS of the device. Some things like connection rates on PIX/ASA aren’t available until 7.x code. I realized last night that it’s OK to have a OID in a template that’s not supported by the device, such as cufwConnRate1udp on PIX 6.3(5). OpenNMS will figure out the OID doesn’t exist on re-scan and not try to poll that OID ever again. Another thing I want to do is change connection limit thresholds based on the type of PIX/ASA license, but I don’t know how I’d do this yet.
Troubleshooting Java is annoying. My first time installing it, I had pages and pages of exceptions and JDBC errors saying it couldn’t connect to the database. It turns out localhost in /etc/hosts was pointing to ::1, so either JDBC didn’t support IPv6 or there was some missing argument. Fixed it back to 127.0.0.1, things started working. That one was truly annoying to figure out. Last night I was trying to add my ASA connection rate stuff and had 593 lines (wtf!?) of runtime exceptions. The “can’t find datasource” error was completely non-obvious that my datasource name was too many characters long. I wanted to dump the jRobin RRD files to check them, but the jRobin inspector is X-based and I didn’t immediately find a way to it from a shell.
The RANCID integration is pretty hot. I like having device configuration details automatically available from the OpenNMS web interface. It took me longer to configure a sample RANCID + viewc (similar to cvsweb) than it did actually turning on the support in OpenNMS. The notification patch for rancid is a nice touch, with it rancid will tell OpenNMS when a config refresh has happened. From the howto, I’m lead to believe that somehow it can manage the .cloginrc password file, but I don’t see how/where this happens. It may not be implemented yet.
So far I really enjoy working with it. It’s taken literally seconds to add new things to check, things that would’ve taken hours of bailing wire and perl to add to my Nagios install. It’s such great relief that interface/service discovery just works. My RTG targetmaker setup is atrocious, takes hours to run, and not the way to do things. I could easily see this replacing the NMS I hacked together. I haven’t dived into notifications yet, so not sure what’s involved there.
Offhand, here’s my to-do and to-figure-out list:
- Graphing of packet loss when doing interface pinging. It sounds like StrafePing monitoring is what I want, which is sort of like Smokeping. The docs warn of extra load, I haven’t checked into this.
- Asset field population from SNMP, including modules/PICs from things like Cisco 6500s, Juniper systems, Cisco CSSes
- ifStatus monitoring of up/down interfaces
- BGP peer state monitoring
- Configure e-mail notifications
- A more sensible graph report structure, i.e. let me click to drill down to more detail about interfaces, not show me octets/packets/errors/discards together (this becomes fail quickly on switches)
- Fetching data from SSH (e.g. log in, run a command, parse counters out of a result, store & monitor the data) or even XML. I need to be able to gather tech-support statistics from Cisco CSMs which is only available via CLI.
- SLB serverfarm/virtualserver stats gathering. Some support for CSS already, but need CSM, Netscaler, and ZXTM too.
It’s interesting that there’s not a lot of blog chatter like Nagios. The bulk of the good information is on the OpenNMS wiki, not much good random info out there. I’m guessing OpenNMS is overkill for a lot of people, and enterprises do their own in-house development to plug into the product. Since it’s all modular and Java, I can easily see it being deeply integrated with other systems and/or do some advanced things. Seeing as how writing my own NMS made me very fluent in perl, working with OpenNMS may finally motivate me to dive in and learn Java.
The future
Oct 25th, 2010 by bwann
The other night I was driving home and in some random apartment window I see two girls with a baby. One was holding/rocking it while she was talking to the other girl. It made me think that’s a lifetime memory in the making that will be forever repeated to the child. “Oh I remember how I used to hold you when you were a baby and I’d talk about Wife Swap with Jill.”
IPv6 in the home: handling DNS/name resolution
Oct 21st, 2010 by bwann
If you’re like me, you have a home LAN filled with various contraptions, computers, mobile devices and the ubiquitous server in the closet. Usually there’s no DNS zone containing your home devices, if anything, maybe a static /etc/hosts entry on your desktop to give a short hostname to your closet server. Otherwise, people just use the IP address of the device. Chances are if you’ve setup a home network, memorizing the subnet (e.g. 192.168.130.0/24) is pretty easy, it’s just a matter of knowing that .10 is closetserver, .15 is your XBOX, and so forth. You just “ssh 192.168.130.10” and you’re good.
When devices have IPv6 addresses, using the IP address isn’t feasible anymore. There has to be some sort of name lookup mechanism. It’s just too long to remember an IPv6 address and there are fiddly things to watch for. Nobody is going to run a real DNS server in their house either.
In the simplest case of using only link-local addresses, you’re just not going to remember your closetserver is fe80::216:e6ff:fed8:d122. You can put a host entry on your desktops, but this involves updating everything by hand. Link-local addressing also presents an additional challenge in that if you have a computer with multiple interfaces (think: laptop with a gigabit LAN interface and a 802.11 wireless interface), you have to specify the interface to use in your /etc/host entry, i.e.
fe80::216:e6ff:fed8:d122%en1 closetserver
This would tell your application to use the en1 (wireless) interface to get to closetserver. But, if you turn off your wireless and plug into your switch, all of a sudden en1 is the wrong interface to use. Now you have to go edit /etc/hosts again to change from %en1 to %en0.
A slightly more advanced example is if your home network uses your router (in my case an Apple Airport Extreme) as a 6to4 gateway. The IPv6 address is still to long to remember, but now the network prefix can change based on your externally visible IP address. Today closetserver may be 2002:4770:2e84::216:e6ff:fed8:d122. Tomorrow your cable modem may reset, giving you a different dynamic IPv4 address and thus a different 6to4 prefix. Now closetserver’s address has changed to 2002:4778:1156::216:e6ff:fed8:d122, and you need to go update all those /etc/host entries.
Solution: Multicast DNS (mDNS). Also known in some documents as “serverless DNS”.
It turns out there’s at least already one solution for all this running behind the scenes that I wasn’t aware of, Multicast DNS. This part of what Bonjour/Zero-conf networks are built on. Each device joins a multicast group on your LAN, advertising and responding to name lookups without contacting an external DNS server or /etc/hosts.
Almost all Apple OS X and iOS devices already have Multicast DNS (mDNSResponder) running, which is how things like iTunes sharing and streaming work, and lets me access resources on the network via name such as “firecracker” for my desktop without knowing the IP address. On Linux, this is handled by Avahi. On Windows, it’s Bonjour for Windows.
This isn’t strictly a problem with just IPv6 either, and there are probably other things out there that do this. This is most interesting to me because the majority of my devices are Apple products.
Here’s how it works:
From my laptop, I want to ssh to closetserver (which is running Avahi). In order to use mDNS, I need to use a “.local” suffix:
lapdance:~ bwann$ ssh -6 closetserver.local
My laptop queries the mdns multicast address ff02::fb on port 5353 asking anyone for a AAAA record of closetserver.
16:01:20.585290 IP6 lapdance.local.mdns > ff02::fb.mdns: 0 AAAA (QM)? closetserver.local. (30)
The mDNS responder on closetserver responds to the multicast group with the AAAA record to use.
16:12:12.098910 IP6 2002:4770:2e84::216:e6ff:fed8:d122.mdns > ff02::fb.mdns: 0*- [0q] 2/0/0 (Cache flush) AAAA 2002:4770:2e84::216:e6ff:fed8:d122 (80)
Then the ssh program initiates the usual TCP with closetserver and we’re done:
16:16:33.518129 IP6 2002:4770:2e84::216:cbff:fe09:a76e.60425 > 2002:4770:2e84::216:e6ff:fed8:d122.ssh: Flags [S], seq 832255505, win 65535, options [mss 1440,nop,wscale 2,nop,nop,TS val 1053744031 ecr 0,sackOK,eol], length 0
Now if I change from wireless to wired, change my external IPv4 address, or even use a new desktop, I can still use the same hostnames without changing anything. Neat! This would also be useful for ad-hoc networks as well.
Men & Mice has a great presentation describing Multicast DNS.
Normally I don’t think a lot about American culture, but the other day I realized just how profound the 60s was on America and the world. I’ve been listening to a Beatles station on Pandora for a couple of weeks, and the other day while working I was keeping an ear tuned to the lyrics as songs played. A couple of the songs were about Vietnam and the draft, and it got me thinking about all the events in the 60s as a whole.
On the surface, to me it has always been “oh yeah yeah the 60s”. I knew individually the Cold War, Vietnam War, the space race, the Cuban Missile Crisis, JFK, Elvis, Beatles, hippies, communism fear, and student demonstrations happened in the 60s. Not to mention the spectacular volumes of now-legendary music that was made then. But thinking about all these events as a whole on the same time line, along with the new civil rights laws and drafts made me realize that there was a lot of permanent change going on. We conscripted men to war. We went to the moon. All within a decade. People and a nation were motivated. It seems hard to me that anyone couldn’t get caught up in it all. Anti-war protests not your thing? Then get on board cheering on the Apollo missions! Maybe go to Berkeley to make/experience cool music instead. Watch some Star Trek!
In this way, it reminds me of World War II. It’s easy to think about all the fronts/theaters as discrete wars. You had the Germans moving around Europe, the Russians defending their cities, U-Boat attacks in the seas, then across the globe we were fighting the Japanese. But, all these events happened within a several years of each other in the 40s and were definitely related.
I can’t name many fascinating things that the 70s and 80s gave to us. Porn and energy crises?
