PCI is just an interface to connect IO devices: how the host can enumerate them, some standard registers, standard mechanisms for access to memory, etc. Any particular IO device will implement a set of standard PCI registers, along with some to do its actual purpose. A network interface may have a set of registers for writing to a transmit ring, for instance, which contains descriptors for packets to be transmitted. (And presumably a similar ring for received packets.) Disk interfaces are often fairly similar in structure.
TOE cards that you mention will contain a complete embedded system (CPU, small OS, etc). They still probably interface to the host using a mechanism like tx/rx rings - the main difference is that the TOE card may have offloaded checksumming, reassembly, etc. (Maybe even protocol stuff like acks.) These cards never made that much sense, since embedded processors are quite power-limited and usually slow, and embedded memory is limited, and it's a disadvantage to have this foreign *system* pretending to just do networking. How quickly does its firmware get updated? (It's not as if a TOE card actually handles all the networking - the host still needs to understand its IP, neighbors, etc.)
A better approach is for the card to simply provide some simple hardware assists to reduce effort on the part of the host. For instance, checksumming - most non-entry-level NICs do this today, and it works great (fast, efficient, well-supported). (I wouldn't be surprised, though, if simply rethinking the packet interface (rings, etc) wouldn't also make a big difference. Consider that NVME is mostly just a totally hot-rodded version of AHCI (with special attention paid to the register overheads of each operation, as well as queue sizes.) Adapting the NVME-style interface for a high-performance/low-overhead NIC seems obvious...)
|