After the recent successes of Space X with The Dragon Heavy Launch vehicle, I thought it would be interesting to understand how the software that is used in these systems is developed.

After a little Googling, here are some interesting findings, sources for the data are Quora, Reddit AMAStack eXchange, GDC 2015/2016

What languages [and] environments does SpaceX use?

  1. Examples include: Python, C++, Go, vim, bash, GNU screen, LabView, and a lot of internally-grown technologies.
  2. They work on everything from large-scale web applications to tiny embedded computing platforms.
  3. They build tech stacks on C#/MVC4/EF/MSSQL via REST to Javascript/Knockout/Handlebars/LESS, C++/Embedded Linux, Python, LabVIEW… which all together enables them to build, launch, and monitor stuff that goes to space.

How many software teams are there?

There are four software teams:

  1. The Flight Software team is about 35 people. We write all the code for Falcon 9, Grasshopper, and Dragon applications; and do the core platform work, also on those vehicles; we also write simulation software; test the flight code; write the communications and analysis software, deployed in our ground stations. They also work in Mission Control to support active missions.
  2. The Enterprise Information Systems team builds the internal software systems that makes spacex run. We wear many hats, but the flagship product we develop and release is an internal web application that nearly every person in the company uses. This includes the people that are creating purchase orders and filling our part inventory, engineers creating designs and work orders with those parts, technicians on the floor clocking in and seeing what today’s work will be per those designs…and literally everything in between. They leverage C#/MVC4/EF/SQL; Javascript/Knockout/Handlebars/LESS/etc and a super sexy REST API.
  3. The Ground Software team is about 9 people. They primarily code in LabVIEW. They develop the GUIs used in Mission and Launch control, for engineers and operators to monitor vehicle telemetry and command the rocket, spacecraft, and pad support equipment. They are pushing high bandwidth data around a highly distributed system and implementing complex user interfaces with strict requirements to ensure operators can control and evaluate spacecraft in a timely manner.
  4. The Avionics Test team works with the avionics hardware designers to write software for testing. They catch problems with the hardware early; when it’s time for integration and testing with flight software it is typically a working unit. The main objective is to write very comprehensive and robust software to be able to automate finding issues with the hardware at high volume. The software usually runs during mechanical environmental tests.

How Big is the current Code Base?

  1. The vehicle code is ~200,000 Software Lines of code (sloc)

What Operating System and hardware is used?

  1. Operating System = Linux
  2. Hardware =  Intel x86 for flight controllers and Power PC for hardware controller
  3. This allows a single workstation to simulate every controller and processor. Hence allowing for automated testing en masse.
  4. Falcon 9 has:
    1. Three flight controller strings on the fist stage and three on the second.
  5. Falcon Heavy has:
    1. Twelve flight computer strings on lower stages.
    2. String cores run two instances of Linux and the flight software, one on each core, on the dual core cpus.
  6. “The triple redundancy gives the system radiation tolerance without the need for expensive rad hardened components. ”
    1. Most flight control systems are triple redundant for reliability (“triplex”).
    2. The use of radiation hardened components is not needed for a suborbital FCS system like that used on Falcon rockets as the flight control is not exposed to enough radiation over a long enough period to induce a fault in the processor, bus, etc.
    3. System that are on-orbit or used for deep space control would generally use rad hard silicon on insulator or silicon on sapphire processors like the hardened PowerPC

Does SpaceX implement continuous testing?

  1. Yes, the goal is to have a code check-in flight validated in a single day.

What is the typical background of a SpaceX Developer?

  1. Game designers have the culture fit they’re looking for.
    1. Game designers typically need to optimize their games for real time or near real time performance,
    2. They must utilize the resources given as best as possible, and must develop and deploy quickly.

How does the Test team determine how to improve the quality of the system?

  1. They developed all the tools that gather data from static fires, engines tests, etc.
  2. These tools give the engineers easy access to the data they need to make informed engineering decisions on the health of the vehicle and for redesigns.

How does SpaceX achieve high reliability for the system?

  1. SpaceX uses an Actor-Judge system to provide triple redundancy to its rockets and spacecraft.
    1. The Falcon 9 has 3 dual core x86 processors running an instance of linux on each core.
    2. The flight software is written in C/C++ and runs in the x86 environment.
    3. For each calculation/decision, the “flight string” compares the results from both cores.
    4. If there is a inconsistency, the string is bad and doesn’t send any commands.
    5. If both cores return the same response, the string sends the command to the various microcontrollers on the rocket that control things like the engines and grid fins.
    6. SpaceX engineers perform what they call “Cutting the strings” where they randomly shut off a flight computer mid simulation, to see how it responds.
  2. The microcontrollers, running on PowerPC processors, received three commands from the three flight strings.
    1. They act as a judge to choose the correct course of actions.
    2. If all three strings are in agreement the microcontroller executes the command, but if 1 of the 3 is bad, it will go with the strings that have previously been correct.
    3. The Falcon 9 can successfully complete its mission with a single flight string.
  3. The triple redundancy gives the system radiation tolerance without the need for expensive rad hardened components.
    1. SpaceX tests all flight software on what can be called a table rocket.
    2. They lay out all the computers and flight controllers on the Falcon 9 on a table and connect them like they would be on the actual rocket.
    3. They then run a complete simulated flight on the components, monitoring performance and potential failures.