Cisco’s first router: Revisiting the venerable AGS+ in 2021
While the AGS was Cisco’s first router, it wasn’t its first product.
The company (“cisco Systems, Inc.”, no capital c initially) had been founded in 1984, the first product launched one year later was the MEIS. (Massbus-Ethernet Interface Subsystem, used to interconnect DEC mainframes).
In 1986 it launched its first router, the AGS multi-protocol router. Take note of the “multi-protocol” moniker, as back in the late 80’s TCP/IP was already present on a wide range of equipment but other protocol-families proliferated until mid/late 90’s. (IPX anyone?)
The AGS was also the foundation for the world-renowned CCIE (Cisco Certified Internetworking Expert) track that was created in 1993 to counter the “cram and forget” certifications in existence.
In contrast to a mere paper exam, the CCIE required you to pass a written exam as a pre-requisite to be admitted to an intense two-day hands-on lab (Reduced to a single day since 2001).
Apart from that, the Internet Backbone within multiple countries was built on the AGS. From my research, this was the case for Switzerland and New Zealand at least (See further links at end of this post if you want to read more about that).
Given the above, the AGS carries a lot of historic significance for Cisco, the Internet and the networking industry as a whole.
That led to me being on the lookout for one of these units, apart from the fact I’m collecting and restoring old electronics as a pastime.
In September of 2020, an AGS+ (Second iteration of AGS, sold since 1989) showed up in the classifieds.
Beyond that, the seller was located in proximity necessitating only a one-hour drive.
I picked up the unit without even checking for its function, which I then did in the comfort of my home.
That’s when a bit of disappointment set in: My AGS+ would only spin its fan, nothing else: No LED activity and no console output either.
The fan in these units is an absolute beast, Cisco used a grossly oversized unit to slim any chance of overheating.
The nickname “leaf blower” suits well – It’s pushing the insane volume of 152CFM (or 4.3m^3/min) and draws 90WAC directly from input (Fan is wired to mains in parallel to the routers PSU itself).
In my case, the fan is the 230VAC version and is an industrial-grade unit produced by EBM-Papst in Germany.
The AGS+ chassis itself has nine slots, where five of them feature access to two backplanes (“slow” Multibus backplane on 1-9, and “fast” CiscoBus backplane in addition on slot 5-9).
The topmost two slots had to be populated by the environmental monitor card (CSC-ENVM) and the processor card, leaving you with 7 free slots for Multibus cards.
The “slow” chassis-wide Multibus has a capacity of 155-Mbps.
The “fast” CiscoBus backplane introduced with the AGS+ improves this more than three-fold to 533-Mbps. To use this backplane, you require a dedicated Cbus-Controller and interface cards which my unit doesn’t have.
My AGS+ was equipped with the following cards:
- Environmental monitor (Also houses 64K NVRAM for config-storage, backed by NiCad-cells)
- Processor card CSC/4 based on 25-MHz Motorola 68040 CPU and 16MB RAM
- 2* Dual Port Ethernet Card (CSC-2E)
- 2* Quad Port High-Speed Synchronous Serial Card (CSC-4T)
With the aforementioned internals, there are four Ethernet and 8 Serial ports located on the back of the device.
The physical termination is provided by separate “brackets” called appliques. In case of the four Ethernet connections, these are AUI-15 interfaces that require installation of an active component called MAU (Medium attachment unit) in order to provide the RJ45-style 10BASE-T connector that you know and love.
In order to install a new interface card you had to number it on the Multibus using DIP-switches, install it in the chassis, mount the appliques of your choice in the back and route the ribbon cables to the new card.
The NiCad-powered NVRAM on the environment monitoring card means that the router can preserve its config for a maximum of 30 days in an unpowered state.
Also note the lack of flash storage for IOS images themselves – This data is stored on EPROMs (not EEPROMs) on the processor card.
This allows you to boot locally, but also means that you can’t update IOS remotely. A SW-upgrade entails prying the EPROMs from the CPU board, erasing them under UV-C and rewriting them in an EPROM-programmer.
Only to allow them to be put onto the processor board again, minding the correct orientation and sequence of EPROMs.
To alleviate this Cisco offered a specialized memory card (CSC-MC+) that featured 4MB of EEPROM flash storage, which allows you not only to remotely update IOS but also has Zeropower RAM fitted as updated media for NVRAM storage, meaning your config is preserved as long as the IC-integrated Lithium battery has power.
This small add-on card is mounted on top of the card cage and connects via a ribbon cable to one of the Multibus interface cards, not taking up a backplane slot for itself.
The root cause of the router not showing any signs of life apart from the fan was quickly narrowed down the single, internal PSU having failed.
In these days, there were no redundant PSUs let alone easily accessible ones.
Accessing the PSU requires disassembling the chassis and even then it’s quite a finicky task – You have to screw each wire to the correct terminal block separately while working in a very constrained space near metal edges.
Measurements revealed that one of the main rectifier diodes on the secondary side had shorted. After replacing it, the PSU tested fine on the bench.
This fix didn’t last for long however, and it failed shortly after being completely reassembled.
Aging electrolytic capacitors are causing issues in lots of electronics, and I went with recapping the complete PSU.
I also replaced the safety capacitors near the AC Input, as those were of the notorious RIFA make that already showed cracks in their plastic case and are prone to shorting out.
With the PSU completely reworked it hopefully stays reliable for years to come.
At this point used the AGS+ for multiple hours and it run like clockwork. Even two out of the three 29 year old NiCad-cells seemed to hold charge properly, which is nothing short of wondrous.
The following evening I returned to the AGS+, curious to see if it’d still boot up with my config I put into NVRAM the evening before.
This time however, I encountered another problem: The router would crash immediately during IOS boot.
In an attempt to isolate the issue, I reverted from IOS 11.0(21a) stored on the CSC-MC+ back to 9.0(1) booted directly from the EPROMs of the CSC/4 card, but no dice.
The AGS+ was stuck in a loop of crashing and rebooting.
%SYS-2-INVALID: Free(4): 81AB6 81AB6 81AB6 81AB6 0 0 0 0 0 0
-malloc(4): 35F2E 48B6 79694 7AA34 A206 A230 35F2E 35F2E 35F2E 35F2E
-Process= "Init", ipl= 5
-Traceback= 3832 31F2 61692 7AA3C 79720 7CF68 7AD8A 3FB4
%SYS-2-SMASHED: Smashed block at 2000000, next FFFFFFFF, prev FFFFFFFF, size -1
-Process= "Init", ipl= 5
%SYS-2-INVFREE: Invalid free list
-Process= "Init", ipl= 5
-Traceback= 3204 61692 7AA3C 79720 7CF68 7AD8A 3FB4 1CFE
Exception: Software forced crash at 0x2AA2EThe HW diagnostics (mainly RAM) run from BootROM didn’t reveal an issue either:
>t m
Memory/Bus diagnostic
Starting Address [0x1000]?
Ending Address [0x1000000]?
Hex argument for variable tests [0xFFFF]?
Select Tests [all]?
Number of passes to run [2]?
Trigger word for hardare debugging [0]?
Message Level (0=silence, 1=summary, 2=normal)[2]?
Testing addresses between 0x1000 and 0x1000000
Begin pass 0, test 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 End pass
Begin pass 1, test 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 End pass
No errors during 2 passesAt this point, I removed all interface cards but the issue persisted.
That left the Multibus backplane, the CSC/4 processor card or the environment monitor card as suspects – And the hunt for AGS+ spare parts was on.
Browsing Ebay, I found someone selling a CSC/3 processor card (The little brother of the CSC/4, based on Motorola 68020 CPU clocked at 30-MHz / 4 MB RAM).
As the card was shipped from Israel, I had to wait for multiple months for its arrival.
The card arrived undamaged and the labelling of the EPROMs indicated a more recent IOS version, too.
Proceeding to run the AGS+ with the “new” CSC/3 wasn’t exactly successful, either.
IOS complained during initialization that there was “more than one Ethernet interface detected” and entered yet another reboot loop.
From my previous research on Twitter I recalled seeing a tweet of someone that acquired a stack of AGS+ routers in 2018.
As it turned out, that someone was no other than Philipp Remaker – A long-time Cisco-Employee and CCIE#1035.
That’s when I decided to try my luck and send him an email – And soon received a response pointing out that he’d be more than happy in helping getting my AGS+ working again.
We were able to deduce that the CSC/3 is loaded with the wrong firmware (Notice the “ASM” on the EPROM labels) which is meant for Terminal aka Access Servers and not for routers.
As this firmware has no routing functionality, it rightfully complains about having more than one Ethernet interface.
As you can run the 9.x firmware on both CSC/3 and CSC/4, it was a simple job of swapping EPROMs for a quick test.
In order to get the CSC/3 permanently functional, the ASM-ROMs were erased (UV-C) and reprogrammed via an external EPROM programmer to the same contents of the CSC/4 “donor” EPROMs.
Just imagine how cumbersome this way of updating the firmware was – Today we might lament about big firmware image themselves, but that pales in comparison.
Not only did we manage to get the CSC/3 operational, but we isolated the root cause to be not a faulty CSC/4 but ENVM-card.
So does that mean the router still crashes and you need to replace the faulty ENVM-card, hunting for yet another rare spare-part?
Luckily not, as the AGS+ is perfectly fine without the ENVM-card. From what I could gather in the documentation, the ENVM was an additions for the AGS+, the original AGS didn’t have it present.
Probably because of this, the ENVM is not enforced to be present.
To this day I’m extremely grateful that Phillip offered and continues to offer his help.
If you want to read more about Phillip, there’s a great blog post that revolves around CCIE history and why he kept his CCIE active for more than a quarter century.
https://blogs.cisco.com/wearecisco/25-years-later-why-im-still-a-ccie
Other references to the AGS and its significance in the early Internet backbone: