SYS — Hardware Products, Early Stage
Systematic Approaches to Design
Fuse agile with waterfall to deliver high quality hardware products on a predictable timeline. Use systems thinking to break down functional blocks to understand and deliver user-focused, high quality products, on time.
Our team in the 90’s took 13 months to develop our first notebook PC — then took it down to 7 months - from whiteboard to shipping dock — while updating BIOS (pre-UEFI), drivers, chipsets, test programs, mechanical enclosures, thermal countermeasures, FCC tests, spinning up a production line and more. (The industry benchmark was 18 months at that time.)
I’ve been involved in over 150 hardware and systems projects, from being the lead to handling NPI to business development to solutions architecture across consumer, networking, storage, cellular networks, automotive and more. Most were successful, with some failures.
The failures could be grouped into 3 buckets:
- “Visionary” product and project managers. These ones consistently fail to heed advice and believe they are Steve Jobs re-incarnated (without his prior work experience at HP, or the culture, supply chain and funding of today’s Apple) and can bend the laws of physics by exhortation. Physics always wins — all you can do is play with material properties at the most basic level, then with the silicon you have. Since experiencing these types, I never work with “visionary” founders or PMs, just with customer driven leaders. [1]
- Failure to think about all the metrics. My first big failure was a timing delay failure with some complex logic. That never happened again. One of my most embarrasing failures was not factoring in the voltage drop of a semiconductor relay in a design where I was asked to replace mechanical relays with solid state systems. I had factored in power, control signals, the control logic, etc. I did all the complex stuff — but failed to check the drop. I’ve never made that mistake again. This stuff happens, and will happen. But, with integrity, the process can be improved continually.
- A lack of integrity. I was in an NPI meeting where I was responsible for production test, and not able to find ground pins on a brand new imaging ASIC. Turns out the lead engineer had turned off DRCs, and got away with it, all the way to tape-out. To this day I have no idea how he pulled it off. The company closed down. Another example; certain radio control features for an IoT system were turned off against my direct advice, leading to excess network congestion and multiple stored-energy-exhausting retries — all coupled with a mis-oriented antenna and a bad power supply. Here too, integrity failures came up, combined with arrogance. Vanity metrics and glossy powerpoint were used instead of basic measurements. Here too, the company closed down. I see this happen too often. I think this is the biggest challenge I see.
You can work around a lack of experience — but integrity issues cannot be worked around. Pick a dependable team above all — even at the risk of slightly less experience or perceived skill.
Try to avoid these kinds of mistakes by having multiple trustworthy eyes on a project and re-using checklists as much as feasible. The key is to think through possible failures, add to your DRCs, and apply relevant DRCs at every stage that matters, whether with a Post-It or a complex constraint file applied to your board layout or FPGA design, or logic test suites applied to your VIP that you are pulling into your ASIC. (This work is critical when thinking of FMEA and traceability as well.)
For industries that are litigation prone, or come with higher liability — mil-aero, automotive, medical, industrial, … — more formal tools and methods are needed. The specific tools and methodologies used for traceability will impact your costs and schedules. [2]
Dealing with regulators is NOT agile CI/CD friendly — you will need to think in terms of formal cycles and releases - and traceability to documented features, test analysis and test case results.
Quality in Blunt Terms
When I worked at Mitsubishi Electric in the late 80’s, I had to sign my design documents as finished. If I wasn’t ready to sign my name, it wasn’t ready. And my boss had to sign too. If he wasn’t ready, it wasn’t. There was a formal meeting where we both signed, in front of a panel of more senior engineering managers. This was a big deal and really stressful.
That leads to 2 interesting words in Japanese:
- Seihin — basically, product. Stuff you see as “Amazon recommended” etc. Lots of stuff you can buy on Alibaba and rebrand on Amazon. If that’s you, my other postings in market entry / beachhead strategy would be more useful than this. [3]
- Kanseihin — finished product. It has all you have to give at that time, in the product. It is polished, and works as promised, if not better. It is designed to slay your competitors with its sharp edge and focused quality in every dimension that matters. Today, we often associate Toyota or Apple quality here — earned over multiple releases of product, over decades. But, there is no reason you cannot aim to Kansehin status — even in a startup.
“I ship when I know we’re at “kanseihin” status at this point is time” should be the mantra of the Founder. You move the bar up over time, but at each moment in time, it is as high as you can go with the knowledge and resources at that time.
Quality Comes with Metrics
Test coverage, reliability, measured ability to handle environmental constraints, response times to inputs, accuracy of inputs, production yield, mechanical tolerances and fit, thermal management … are all measureable factors for any hardware product. Aesthetics are also measureable in terms of user ratings and response.
Before you invest much time into hardware development, even if you are using an off-the-shelf board to start with, take some time to identify the critical metrics you need to track. It can be on a Post-It in the worst case, or in a formal document. The metrics can also be staged from prototype to release candidate to final release — but know what the metrics are. Only ship when critical metrics are met.
- You can ship early versions — as long as your customers know it is an early version, and you update it later. You can call it a beta program if you like. But be clear about the status and the transition.
- Kanseihin does not mean cost/time over-run, or putting all possible features into a product release. On the contrary, it focuses on reducing the feature set, so that all that is delivered, is excellent. Less is more.
System Boundaries and Signalling
Whether you are building hardware or software, you have system blocks with clear boundaries, and a method of signalling (information exchange) across the boundaries. These can become fairly granular (output of an op-amp) or large (output of a massive data processing application). You may use spread-spectrum techniques or RAID or HDFS. You might use TCP/IP or something else. Basically, you process and move signals/ data/ information …
Each system boundary is generally defined by a set of input conditions (user clicks, voltage pulses, neural network classifier output … ) a set of environmental conditions (power, available data, …) a process (data conversion, signal processing, …) and a series of outputs. The inputs are transformed into more valuable outputs, which are then inputs to the next stage of processing, or delivered to the user.
If there is no change in value/worth of an output compared to input, then the system block is worthless and should be eliminated or reworked. Values can be audio amplification, or video filters, or other.
Error Handling and Design for Test
An elevator may be designed for 1000 Kg, but cannot suddenly fall and crash at 1200Kg when 2 more people squeeze in. A vision system may be designed to detect pedestrians, but cannot suddenly stop working because a cow came into view, or a childs ball bounced onto the street.
How these “corner cases” are handled are important. Graceful degradation of function is key. Think about what could happen, then what should happen. How is the rest of the system notified?
3D 2X Model
This is what I call my approach, with 3 basic phases, Define/Architect, Develop/Make, Deploy/Support, executed in parallel, across Engineering and Business. If you are a solo founder, this still applies. [3]
Each phase is iterative, and takes capital, which aligns with hardware funding cycles, whether internally funded or venture funded. The phases themselves are also iterative — a loop with nested loops.
As each iteration proceeds, more and more of the product features need to get locked down, leaving a few to the last moment. Don’t spend too much time on packaging and box contents in the early stages — instead focus on core decisions like user functionality, computing platform, protocols to be supported, remote support hooks and more. Get the core needs understood and stable early, then polish up as you progress.
Changing your compute platform a week before shipping is a non-starter — but changing the webpage a QR code directs to can be done while your product is in transit. Think in terms of a freeze-sequence. As you repeat a loop, the iterations in the nested loops reduce.
Define/Architect Phase.
Focus on locking down the big items — the CPU, the OS, the form factor, basic UX, target users and environmental needs. Minimize the number of radios to ease global certifications, and use pre-certified radio modules wherever possible. (When you are bigger, you can go direct to chip on board and save the margin stack; in the early stages, get the product out first.) Creating simple prototypes that can be used to explain ideas to your stakeholders is helpful — from UIUX models to lego models (illustrated below) to hand drawings.
The goal is to make sure you’ve captured real needs, and your stakeholders can imagine what the outcomes could be — before spending cash on prototypes.
At this point, an early budget should be forming — you should begin to have some idea of what you are going to build, how much effort will be needed, what kind of parts and their costs, etc. Bringing in someone experienced for a short time at this phase will help generate more seasoned estimates. You will need this information for an early stage pitch deck for internal or external investors.
A clear listing of customer needs against technical specifications is the first step in traceability. This also helps in project planning — even if you are a one-person startup. Expect fine details to change later.
Develop/Make Phase.
The business team should be using the assets they got in the first phase, to continue to build sales collateral, and build out the sales pipeline. In a startup, this is *also* the Founder’s job.
While this critical business work is ongoing, the technical team should be making rapid prototypes of the entire product. This is key — the software team will need time to build out a product and have it tested — and the business team must leverage collateral to lock in customer contracts.
This work may force changes in hardware, which could range from additional on-board storage to faster CPUs, hardware encryption due to updated customer needs and more. Try to minimize changes, and plan for upgraded versions later if possible.
During this phase, you’ll generate a Bill of Materials (BOM), schematics, PCB layout, enclosure design and tooling files, a tentative manufacturing process document that details how you product comes together, a test plan to confirm that you know what to test, how to test, and what metrics to track. This will get you a way to estimate your Cost of Goods (COGS) and a supply chain analysis will inform you as to how long it will take to build and ship a product, and what the minimum order quantity (MOQ) will be, and the following step changes in quantities that drive down costs.
From this information, you can build out your cash flow needs that reflect your working capital needs. (The original dorm-room genius at DELL was to avoid working capital needs, and to instead get paid first, build a PC from standard parts, then ship.) Find ways to reduce working capital — smaller lots, vendor financing, customer financing, as early as possible. The less cash you need, the faster you become profitable. If you are a startup, it also means less dilution.
From this, you can now build an even more accurate dashboard that tracks time to zero-cash above all else. Companies die when they run out of time.
Deploy / Support Phase
Here, we move to production and selling the product; repeated across Proof of Concept, Prototype, Release Candidate and Production Release.
During each iteration, you and the company are learning; you can think of each cycle in terms of a Cycle of Learning as well.
You are learning about what it takes to create the product, verifying function against your internal engineering metrics, as well as user feedback and market metrics. A key part of this phase is Support.
Moving to a production setup needs supply chain planning, production sequencing, quality checks, packaging, labels, compliance checks, contracts, commerce integration and much more. Some of these issues can be outsourced, but there needs to be awareness of the issues and related metrics.
Try to keep production in-house for as long as possible; that way design and production are co-located and opportunities for learning and improvement are enhanced. Over time, scaling will force separation, which will then force more formal communication and dashboards.
Support needs to be timely, accurate, useful and as seamless as possible. (I have yet to have a single support issue solved by a chatbot — keep those for collecting information about the customers needs/interests, and be honest about that. )
Support also implies remote access, OTA updates, remote diagnostics and ability to match machine IDs to specific customer accounts.
The easier you make it for your customer to inform you of their needs, and for you to act usefully on that, the more detail you can get.
Support should not be thought of as a cost center, to be banished to bot-world or the lowest-cost call center you can find. Support is your customer-funded knowledge center. Turn every support call into an opportunity to learn more about your customers.
From support requests, you will gain knowledge about how your product is actually being used, and what customers really want. This is the largest source of customer-funded market insight you can get, in real-time.
As you get larger, you will need to deploy classifiers, self help systems and other methods to process customer feedback and provide help. But that only happens if you have enough customers — which means you went through Cycles of Learning and delivered a meaningful product to Kanseihin standards in a timely manner, and then took the Share of Market (SOM) that you deserved.
Good luck!
[1] https://manu-pillai.medium.com/visionary-founders-and-reality-939edfa0e6e7
[2] Consider tools like Polarion (from Siemens) and Modelio from https://www.modelio.org — but be aware that these do not (as Jan 2022) integrate naturally with Cadence, Mentor (Siemens) or JIRA/Confluence. One of these has to be the Single Source of Truth, with additional work needed to bind these flows together. This is significant work to ensure traceability and compliance.
[3] https://manu-pillai.medium.com/battle-for-som-pt-1-baad2e44560b is the first in a series exploring Share of Market thinking along with funding.
