This is a collection of my reminiscences about the Michigan State University Computer Laboratory's Systems group, aka Systems. I was by no means one of legendary greats like Dave Dunshee, Bob Bedoll, John Renwick, and Doug Nelson, so these notes are more oriented toward capturing memories of the times, rather than detailing my modest accomplishments. See also Systems People.
Subsequent to the initial release of this document, I received several helpful comments from other Systems folks, which I have incorporated. Direct quotes are in italics. But more importantly, a Wikified version of this document is now available at: http://msusystems.pbwiki.com/SystemsReminiscences. The Wiki version will probably be more up-to-date than this version by the time you read this.
I worked for Systems from 1981 until its effective dissolution in 1988. (I remained at the MSUCL until 1995.) I came to Systems in a very unusual way: I was first an employee of the MSUCL User Information Center (UIC, aka User Services). Sue Gossman (Dunn) and I were among the very few people to come to Systems in this way. I did hang out with Peter Poorman, Bryan Koch, and a few other Systems types before joining Systems. I probably met them through the very sociable Rich Wiggins, a colleague at UIC.
Each summer, the Systems group conducted a grueling non-academic class called Systems Class. Nearly all of the attendees were newly hired undergraduates who wished to become systems programmers. As I understand it, applicants were interviewed by Richard Moore, and if they passed muster, were hired for a few weeks to take the class. Those who passed the class were generally offered longer-term part-time employment; those who didn't pass were let go.
My understanding was that in the 1970's, the Systems group was one of the very few places on campus that offered challenging, paid programming employment to students. There were quite a few applicants to Systems class in those days, and the quality of the applicants was high. In later years, there were more programming jobs to choose from, and Systems could not afford to be as picky. [Can anyone confirm these memories?] Toward the end, all applicants were accepted, and anyone who completed the class without dying of exhaustion was hired. I believe that even in the early years, few students were actually fired at the end of the course; most students who couldn't hack it figured it out for themselves and dropped out voluntarily.
The class consisted of two very intensive weeks of lectures and assignments, followed by about two more weeks of projects. The first two weeks were like boot camp: not only were days filled with lectures, but there were non-trivial assignments every day, and many students pulled near all-nighters day after day.
[KDH: One of the projects students worked on in Systems Class was the infamous Console Driver project. It got students feet wet with PP programming. Actually, it got them thoroughly drenched. :-) Anyway, it was a stand-alone PP program which displayed some text and hopefully, if it worked, did some interaction with the keyboard. It was loaded by a special deadstart loader which was provided. IIRC, that deadstart loader could select a student program for testing out of several on the deadstart tape.]
Class was conducted by one or two Systems full-timers, with others giving guest lectures. The entire operating system was described in some depth. The STRAPs (Systems, Tasks, Responsibilities, and Procedures) documents were discussed at length, with special emphasis on STRAP 9, which described coding standards.
I audited Systems class in 1980, as a full-time UIC employee. It was very unusual for someone to audit the class; normally all attendees were trainees whose continued employment theoretically depended on doing well in the class. In my case, it was merely my reputation that was on the line, since at the time I was not seriously considering defecting to Systems.
The 1980 Systems class was taught by Nora Gaty and, I think, Bryan Koch. Nora was one of the few full-timers who did not come up through the ranks, and I was surprised that she would be teaching the class only a year after coming to MSU. One of the guest lectures I recall was by Mary Kestenbaum, on PP Resident, aka STL. This was a tightly-coded subroutine library that resided at locations 100B-777B in most PPs. She had recently rewritten most of the library to use new structured programming macros, and had saved a few bytes in the process. She handed out a listing of STL, which I kept as an example of vital SCOPE/Hustler code. I was distressed when it went missing many years later. Fortunately, Ken Hunter had saved the complete source to SCOPE/Hustler, so I was reunited with STL around 2004.
By 1981, I was getting tired of answering the same questions over and over again as a consultant at UIC, so I did defect to Systems.
Most of Systems was housed in 301 Computer Center. This was probably the only office in the building protected by a CypherLock, an electronic 4-digit combination lock. (The machine room also had CypherLocks.) The combination was changed fairly often--perhaps monthly. [Does anyone remember how often?] Systems employees were not issued keys [?]; they had only the combination. The CypherLock ran from building power, so during one of the not infrequent campus power failures, employees were locked out and had to pound on the door to get someone to let them in.
The door to 301 was to be kept closed and locked at all times. The reason for the high security was presumably the protection of the mountains of paper listings kept on shelves around the room. Some of SCOPE/Hustler's security came from "security by obscurity", and a careful reading of some of the listings might have turned up the secret HAL passwords, or revealed previously undetected security flaws.
There was a receptionist's desk just inside the main door in 301. Throughout much of Systems' history, Technical Assistant Janice Parnell (nee Snyder?) sat at that desk. (In earlier years, Jan had been known as "Candy", but this nickname had largely gone extinct by the late 1970s.) Jan answered the phone and did some filing. She was reluctant to do keypunching or typing, for fear that her position would downgraded by Human Resources if they found out.
301 had a back door, where there was an old refrigerator, and what must have been one of the first microwave ovens ever made. In the 1970s through at least 1987, there was some sort of a soda pop club, known as the Coke Fund. Some employees--Larry Kingsbury might have been one of them--periodically went to the store to buy large amounts of Faygo (?) soda in glass bottles. In later years, we made the switch largely to Coke. The pop was then sold for some low price. [Does anyone remember the price?] Members of the club checked off a box on a printout taped to the refrigerator door. Occasionally the checkmarks were counted and the members were billed. An unusually voracious pop drinker was James Taylor, whose doctor had advised against the drinking of tap water due to JET's glaucoma.
Systems was filled with Herman Miller partitions. Full-timers had their own cubicles, with multiple work surfaces and many shelves. Students generally shared a large common area known as the Student Ghetto; each had his/her own desk and shelves. The shelves were filled with listings and pop cans.
[KDH: I remember some full timer's offices featured empty pop bottles, too. It wasn't only the students. There was also an empty coke bottle stashed in the Machine Room (!!) the whole time I was there (1984-1987). I have no idea who left it there. Might have been someone in Operations. For all I know, it might still be there, even though the CYBERs are long gone.]
The Herman Miller walls came in highly tasteful 1970s colors. Most cubicles were personalized by their inhabitants with posters and other paraphernalia. Some walls were solid and would hold a push pin easily, but most had a layer of soft foam under the cloth covering. Ordinary push pins often fell out. I got around this by buying extra-long pins.
Room 318, across the hall, housed the Systems Integration group. This room did not have its own CypherLock.
The single most defining characteristic of the Systems group was probably the arduous code review process. Each source code modification was printed out, and stapled to a code review form called a "gift certificate". The listing was submitted to the coder's group leader for first level code review. (If the coder was a group leader, the listing was submitted to another group leader.) Then the listing was sent to a second group leader, and finally to Richard Moore, the manager.
At each step, the reviewer was expected to very thoroughly inspect the code for both correctness and style. Then the code was either Rejected, Accepted if Comments Corrected, or Approved. If the code was rejected, it went back to the coder (and any previous reviewers). Otherwise, the listing was sent to the next level of the review process. Richard Moore did in fact review code very thoroughly, and if he caught something that another reviewer missed, that was considered a black mark against both the reviewer and the coder. And Richard was very finicky.
Speaking of colors, Richard Moore had standard colors for his corrections. Other reviewers werefree to use their own colors. I believe that one color was for mandatory corrections, and the other was for suggestions. RRM's colors may have been green and orange, and he always used felt tip markers. But I have a 1984 gift certificate on which TDD used an orange felt tip marker, apparently usurping RRM's color. This would surely be a firing offense. Does anyone remember the standard colors?
As a result, it was almost unheard of for a code submission to be marked Approved by anyone on its first pass through review. Every reviewer felt compelled to find something wrong with the submission, to avoid being considered a pushover or an easy grader. In one infamous case, I had a submission rejected because I used the personal pronoun "he" IN A COMMENT in the code, referring to the user of the program. The correction was that the comment should read "she or he" or "s/he".
Larry Kingsbury had a way of dealing with this. He would prepare his initial submission with some deliberate innocuous errors that would satisfy the reviewers' need to "find something". Then on the next pass, he'd submit his original version that did not have the errors.
Though the code review process may have been overdone at times, systems programmers were nearly unanimous in their agreement that it was much better than the "anything goes" approach sometimes used by Control Data and others.
The manager of the Systems group was, of course, Richard Moore. Under Richard were several groups. The number and identity of these groups changed with the times and the personnel. Each group had a group leader and typically 2-3 full-timer programmers and 0-2 student programmers. There were few clerical employees: just Jan and sometimes a student.
One group that remained throughout most of System's history was the System Integration group, SysGen. This group's mission was to collect source code modifications and create new deadstart tapes. Nowadays this would be called "doing a build". There was no standard "make" facility; instead, the System Integration group ran a very complex series of homegrown programs and exec files. These were revised and improved many times over the years. The scripts controlled dependencies so that compilation jobs ran in the right order, and unnecessary recompilations were minimized.
The leader of the System Integration Group was called SysGen. This was the entry-level group leader position, and most group leaders did their time in SysGen. The job wasn't glamorous, but it did provide some management training, and someone had to do it.
Most systems programmers, if they stuck around long enough, were eventually offered a promotion to group leader. Even though it meant a sentence to SysGen, nearly all accepted, partly due to the pay raise involved. Dave Katz was an exception. He was repeatedly offered a group leader position, but always refused it. Most of his colleagues, I believe, respected the guitar-playing DMK for not selling out to the establishment.
[DMK: You'll be happy to know that I have clung tenaciously to the bottom of the org chart for my entire career. The good news is that in Silicon Valley this is not limiting. I ended up in director-level positions at both Cisco and Juniper and have never managed anyone but myself (and not well, at that.) Junior guys that I hired in became VPs and I work for them. But it works well.]
Above Richard in the hierarchy in the 1970's was Assistant Director Lew Greenberg, and above him, Associate Director Julian Kateley. Supposedly above Julian was a shadowy Director named Lawrence Von Tersch. Von Tersch kept his office in the Engineering Building, and rarely came to the Computer Center. Julian actually ran things. As far as I know, I never saw Von Tersch. The closest association I had with him was that one of my housemates, Mary Clark, had once dated his son.
Lew became Director when Julian left for Colorado State, having been lured by the supercomputer there. He remained Director until his retirement in 2002.
Most of the essence of SCOPE/Hustler was completed by the time I came to Systems, though I was familiar with some Systems projects earlier (I started at the MSUCL in 1975). Here are some notes on a few projects.
The "Hustler" in SCOPE/Hustler referred to the integration between batch and interactive service. Early versions of CDC SCOPE 3.x had simply horrid interactive service, with each command essentially starting a batch job whose output was printed at the terminal when the job was complete. Hustler not only implemented true interactive service, but merged the scheduling of batch and interactive jobs in a fair and efficient manner.
By the time I arrived at the MSUCL in 1975, Hustler was pretty much complete. As I understand it, some of the final tweaks to Hustler's scheduler were done by Lamott G. Oren, a student who lived on the same floor of Shaw Hall as I. Lamott, who initials fortuitously were LGO, was apparently one of the most talented students who never become a full-timer. (He went to work for Texas Instruments.) LGO was also the MSU systems programmer who most closely resembled rock star Edgar Winter.
ARGUS was MSU's version of JANUS, CDC's program to control line printers, card readers, and card punches. As enhanced by Valeria Szigeti (Welbeck), ARGUS also took on the responsibility of handling MERIT network traffic. This task may have been given to ARGUS because initially, all MERIT traffic was batch jobs, and a MERIT job arriving at MSU could be treated like a deck of cards.
The source code to ARGUS (written in CPU COMPASS) was extremely difficult to follow, and was virtually free of comments. For a few years, programmers unfortunate enough to have to work on ARGUS grumbled how much worse it was than the rest of SCOPE/Hustler. Finally, around 1973 or so, Systems decided to rewrite ARGUS, using the latest coding and design standards. Top programmers labored on it for what must have been several person-years. Advanced macros were written to allow structured programming in COMPASS.
Several weekends were set aside for ARGUS 2 testing. ARGUS 2 testing was generally popular with the user community, as I recall, because all computing was free during that time. (The Computer Lab charged high rates for computer resource usage at the time.) DMK: I was a high school student in the summer of 1974 getting lots of free computer time when they were doing testing.
After a number of tests, it became clear that ARGUS 2 was simply too slow. The code was a joy to behold, but it consumed too much CPU. ARGUS 2 was ashcanned, perhaps around late 1975, and the original ARGUS remained in production for the rest of SCOPE/Hustler's days. Fortunately, the structured programming macros lived on.
ARGUS 2 was one of the very few failures in Systems. There were numerous modest-sized student projects that never saw the light of day, but none compared with ARGUS 2.
By the early 1970's, demand for interactive use was increasing rapidly. Adding a lot more lines using CDC's muxes would have impacted mainframe performance unacceptably. And CDC's more advanced 2550's were very expensive. This caused MSU to decide to create its own interactive front-end.
Initially (legend has it), a DEC 11/45 was to be used, but the DEC salesman rubbed Lew Greenberg the wrong way. We ended up with an Interdata (later Perkin-Elmer) 7/32, the first 32-bit minicomputer. Lew designed a DMA device for the Interdata that attached to a CDC channel.
Bob Bedoll and John Renwick, reprising their successful partnership on MSU EDITOR, developed a prototype front-end that was called GREASY. This was running by about 1975. The success of the oddly-named GREASY led to the FREND project, which became operational around 1977. Printing support was added, so we could use the less expensive commodity printers with the semi-standard Data Products interface. (Later it became clear that the Centronics interface, not the Data Products interface, was the industry winner.) [DMK: KRJ and JKR and I all worked on the printer support around '78 or '79; I believe I did most of the 7/32 work and John and Ken worked on the mainframe side, but it's all fuzzy; seems like maybe Ken did FREND work as well. We used to come in at 5pm on Saturdays, skip out for SNL (in its first couple of seasons) and then come back and work all night.]
Later, X.25 support was added to FREND. [DMK: John started the MICI (MSU Inter-Computer Interface) project, which was to be a network front-end (another 7/32) for the mainframe. I picked it up when he left. That's where the X.25 code went (I wrote that too, a fact I generally don't admit to these days.) It ended up being the front-end to the infamous Contel broadband network. I handed it off before I left in 1985; I don't remember exactly how it all played out.]
Starting in 2004, I reimplemented portions of FREND as a C program for Windows or Linux, to be used with the excellent DtCyber CDC 6000 emulator; see http://frend2.sourceforge.net.
By the mid-1970s, the 6500 was heavily overloaded. The user community was complaining that much of the machine's capacity was being used by the Computer Lab--and much of that was due to Systems. MSU implemented a stopgap measure around 1977 by buying a used 6400 and devoting it to CL internal use. MSU CL employees were forbidden from using the 6500 using business hours.
Initially, the 6400 was a completely separate machine, with separate peripherals. This was inconvenient, because work done on one machine had to be ported to the other by dumping files to tape on one, moving the tape to a different drive, and reloading the tape.
Systems responded with the Multi-mainframe project, which implemented shared permanent files between the two systems. I believe that it was also possible to submit a file to an input or output queue on the other machine. Synchronization was done via shared ECS.
A great deal of work was done to take advantage of the rather modest 6400 hardware. I believe that Dave Dunshee, Bryan Koch, and possibly the cigar-chomping David Roderick were among those involved. The effort paid off when the 750 was installed around 1979. We ditched the 6400, and instead hooked the 6500 and 750 together.
The "15 control points" project may have been the largest software project ever whose goal was to free up a single bit in memory.
The OS was designed so that when a job was in memory, it was assigned to one of 7 slots named control points. There was a 3-bit field giving the job's current control point number. (A value of 0 for this field had special meaning; there wasn't really a control point 0.)
When we got the Cyber 750 around 1979, with at least twice the amount of memory as the 6500, the amount of memory in the machine was no longer the limiting factor on throughput. Often memory would go unused, as jobs waited for one of the 7 control points to free up. The obvious solution was to increase the number of control points, and change a job's 3-bit control point field to 4 bits.
It turned out that there were many references to 3-bit control point fields through the OS. More recent code used macros for the size and location of the field, so recent code could be changed fairly easily. However, older code had hard-coded constant references that were not always easy to find. Eventually, Systems was to spend several programmer-years poring through the system, finding and changing all control point references.
The project succeeded. The only downside of the new 15-control point system was the console's B display had to be changed to show less information per job, because the screen wasn't big enough to hold all the original information for each of 15 jobs.
The permanent file I/O queue project addressed a reliability issue with the operating system. An unfortunate quirk of SCOPE/Hustler was that queue files were temporary files whose metadata resided only in ECS. (Queue files were queued up input jobs from card decks or SUBMIT control cards, or output files destined for line printers or the card punch.) Frankly, SCOPE/Hustler crashed fairly often, and when it crashed, it needed to be redeadstarted. On a good day, one could do a "recovery" deadstart, and the tables in memory were recovered without loss of data. But if a full "normal" deadstart was required, all queued up jobs were lost. This could be as much as 1000 jobs on a busy day near the end of a term, and it meant that students and researchers had to resubmit their jobs.
The PF I/O queue project was undertaken to catalog input and output files as permanent files. When the jobs ran or were printed, the permanent files were attached and purged. Since the PF system was very reliable, queue files were virtually guaranteed to remain intact even after a bad crash.
The project was initiated by Bryan Koch before he left for Cray, and implemented largely by Tom Davis and Jeff Piper. Tom had very high standards for the project, and no one could meet them--not even himself. This resulted in many cycles of ripping everything up and starting over. In the end, the resulting system worked well.
By 1979, many sites were using "smart" terminals to provide a "full-screen" interactive environment. But MSU was stuck using glass TTYs. In particular, using the editor from a line-oriented interface was rather painstaking. Lew Greenberg (I think) came up with the idea of using programmable terminals to provide a full-screen line-scraping front-end to MSU EDITOR.
While I was still working for UIC around Fall 1979, I was given the opportunity to design and implement the project, which became known as SCREDIT. I toyed briefly with the idea of using PL/M, a PL/I-like language whose compiler Larry Kingsbury and Dave Dunshee--now MSU expatriates at SRI--had ported to the CDC. However, it seemed risky, so I ended up using the same approach as the FREND project: using COMPASS on the CDC to allow me to program in Intel 8080 assembly. Peter Poorman had already written an 8080 cross-assembly COMPASS systems text, apparently just for the heck of it. Some host-based programs, probably including FREND's POST732 program, were used to process the Cyber loader output to create Intel-format binaries. These were either fed to a standard PROM burner, or downloaded directly to the intelligent terminal.
The brand of intelligent terminal was the Ontel. The initial model used during development was the OP-1. The University of Michigan had a number of Ontels, which they used for a similar project. They written had a debugging monitor which, perhaps inevitably given the popularity of the Star Wars movie with its Obi Wan Kenobi character, was named Kenobi. Kenobi was to prove quite valuable during development. The later diskless model OP-1/R was used during actual deployment. Most Systems employees came to have an OP-1/R in their office, and Ontels were used, starting in Fall 1982, to replace keypunches on the second floor of the Computer Center. It now seems incredible that keypunches were still playing a major role in MSU computing as recently as 1982.
Also around 1982, after I had moved to Systems, I ported SCREDIT to the new IBM PC. Jerry McAllister added file transfer, using a highly proprietary protocol he and I designed. Shortly afterward, I implemented DEC VT220 terminal emulation for use with the VAX, and a mode to accommodate the IBM 7171 front-end to our IBM mainframe. I also threw in a little-used mode to accommodate FSE on CDC NOS/VE.
[DMK: I loved SCREDIT so much that when I left for Merit I had you
cut me a version that limited the fractional line numbers to three digits, and
then I wrote a series of MTS macros to implement the subset of MSU Editor
commands it issued. I used it (on the PC) for much of my tenure at Merit
when I worked on MINOS and then the 3270 emulator that we picked up from UBC for
the library card catalog project.
I regularly mention the cool hack you did to implement a vertical retrace
interrupt on the original CGA card (since it was left off of the card,
though hints of it were in the docs.) I believe I helped you get around
the clock-runs-too-fast problem that came after you reprogrammed the single
interval timer to run at 30 Hz instead of 18.2 Hz. [MRR: Yes, that was
a clever trick.] This usually leads into a discussion of why the metric
in Novell's IPX RIP routing protocol was in units of "ticks" and
why a tick was 1/18.2 seconds, and why the clock rate on the original IBM
PC was 4.77MHz even though it had a 5MHz part. We ended up with *very*
jittery "ticks" through INT 0x0C that averaged out to the proper
rate.]
In 2006, I began reimplementing SCREDIT as a C program for Windows; see http://scredit2.sourceforge.net for a rather incomplete version.
One of the last CDC-related projects undertaken by Systems was the port of SCOPE/Hustler to the very low-end CDC Cyber 810. This machine was originally purchased for an ill-fated network authorization project. It found itself replacing the much faster but older CDC 170-750 because its hardware maintenance was cheaper. David Bright began work on porting SCOPE/Hustler around 1985. SCOPE/Hustler ran on the 810 until around 1989.
[KDH: Dave Bright's work in porting SCOPE/Hustler to the 810 included making MSU Deadstart compatible with CIP. CIP was (at least according to CDC) required for Deadstart of 800 series machines, and optional for the earlier machines. At MSU it was used with SCOPE/Hustler on the 750 as well, after SCOPE/Hustler was running on the 810 (previously, Hustler was deadstarted on the 750 and 6500 without CIP). After SCOPE/Hustler was running on the 810, it became the CL development machine, and the 6500 was scrapped. Literally. RIP.]
In August 1988, the MSU Computer Lab underwent a significant reorganization. Productivity and cooperation amongst units had dropped, and in addition, the world of computing was changing. It was felt that the CL needed a kick in the pants. Systems technically ceased to exist as part of the reorganization, though most Systems employees found themselves still sitting in the same offices, still working for Richard Moore.
One could argue that in some sense Systems did survive the reorganization. But with the CDC systems disappearing around this time, and with the role of systems programming having greatly diminished by then, it seems reasonable to declare the history of the Systems group to have ended in 1988.
Like me, many former Systems employees look back with fondness on their days as MSU systems programmers. The specific knowledge is no longer relevant, but the discipline and general skills live on in the many MSUCL Systems graduates who now work in the IT industry.
What also lives on is the source code and internal documentation generated by Systems. See http://60bits.net. SCOPE/Hustler now also runs on PC hardware using software emulation; however, due to licensing restrictions placed on CDC software by the current owner of the copyrights, SCOPE/Hustler cannot yet be widely distributed.
Thanks for contributions from Dave Katz and Ken Hunter.
Mark Riordan mrr (at) msu.edu Written Sept 2006; last updated 24 Sep 2006 09:05:44 PM